Discriminant Adaptive Nearest Neighbor Classification

Nearest neighbor classification expects the class conditional probabilities to be locally constant, and suffers from bias in high dimensions We propose a locally adaptive form of nearest neighbor classification to try to finesse this curse of dimensionality. We use a local linear discriminant analysis to estimate an effective metric for computing neighborhoods. We determine the local decision boundaries from centroid information, and then shrink neighborhoods in directions orthogonal to these local decision boundaries, and elongate them parallel to the boundaries. Thereafter, any neighborhood-based classifier can be employed, using the modified neighborhoods. The posterior probabilities tend to be more homogeneous in the modified neighborhoods. We also propose a method for global dimension reduction, that combines local dimension information. In a number of examples, the methods demonstrate the potential for substantial improvements over nearest neighbour classification. Introduction We consider a discrimination problem with d classes and N training observations. The training observations consist of predictor measurements x = (zl,z2,...zp) on p predictors and the known class memberships. Our goal is to predict the class membership of an observation with predictor vector x0 Nearest neighbor classification is a simple and appealing approach to this problem. We find the set of K nearest neighbors in the training set to x0 and then classify x0 as the most frequent class among the K neighbors. Nearest neighbors is an extremely flexible classification scheme, and does not involve any pre-processing (fitting) of the training data. This can offer both space and speed advantages in very large problems: see Cover (1968), Duda & Hart (1973), McLachlan (1992) for background material on nearest neighborhood classification. Cover & Hart (1967) show that the one nearest neighbour rule has asymptotic error rate at most twice the Bayes rate. However in finite samples the curse of dimensionality can severely hurt the nearest neighbor rule. The relative radius of the nearest-neighbor sphere grows like r1/p where p is the dimension and r the radius for p = 1, resulting in severe bias at the target point x. Figure 1 illustrates the situation for a simple example. Figure 1: The vertical strip denotes the NN region using only the X coordinate to find the nearest neighbor for the target point (solid dot). The sphere shows the NN region using both coordinates, and we see in this case it has extended into the class 1 region (and found the wrong class in this instance). Our illustration here is based on a 1-NN rule, but the same phenomenon occurs for k-NN rules as well. Nearest neighbor techniques are based on the assumption that locally the class posterior probabilities are constant. While that is clearly true in the vertical strip using only coordinate X, using X and Y this is no longer true. The techniques outlined in the abstract are designed to overcome these problems. Figure 2 shows an example. There are two classes in two dimensions, one of which almost completely surrounds the other. The left panel shows a nearest neighborhood of size 25 at the target point (shown as origin), which is chosen to near the class boundary. The right panel shows the same size neighborhood using our discriminant adap142 KDD--95 From: KDD-95 Proceedings. Copyright © 1995, AAAI (www.aaai.org). All rights reserved.